Learnings Options End-to-End for Continuous Action Tasks

We present new results on learning temporally extended actions for continuous tasks, using the options framework (Sutton et al. [1999b], Precup [2000]). In order to achieve this goal we work with the option-critic architecture (Bacon et al. [2017]) using a deliberation cost and train it with proximal policy optimization (Schulman et al. [2017]) instead of vanilla policy gradient. Results on Mujoco domains are promising, but lead to interesting questions about when a given option should be used, an issue directly connected to the use of initiation sets.